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ABSTRACT 

This paper describes the results of an effort to 
apply a classification system developed by the Human Resources 
Research Organization to a set of termanal objectives in reading. The 
classification system contained five factors, of three levels each. 
Some 312 terminal objectives And 766 approximations were classified 
by three raters. Agreement among raters was poor at the start of the 
rating, but improved dramatically with practice. It was concluded 
that: (1) classifying objectives is a useful procedure by which to 
evaluate their communicability, (2) the verb is probably the most 
important aspect of an objective, and (3) reading objectives are no 
more difficult to classify than are objectives in other instructional 
content areas. (Author) 
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llie Human Resources Research Organization (HumRRO) is a nonprofit 
corporation established in 1969 to conduct research in the field of training and 
education. It is a continuation of The George Washington University Human 
Resources Research Office. HumRRO*s general purpose is to improve human 
performance* particularly in organisational settings* through behavioral and social 
science research* development, and consultation. 
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Prefatory Note 



The research upon which the paper is based was done under contract 
with the New York State Education Department. The work was performed 
at HumRRO Division No. 5, El Paso, Texas (now part of the Western 
Division). Dr. William H. Melching was the Principal Investigator. 
Drs. Paul G. Whitmore, Edward W. Frederickson, and John P. Fry served as 
raters. Mrs. Dorothy A. Alderman performed the numerous tabulations of 
rater judgments. 

This paper was presented at the American Educational Research 
Association annual meeting in Chicago, Illinois, April 1972. It was part of 
a Critique Session entitled "Functional Literacy." Dr. Vhomas G. Sticht of 
the HumRRO Western Division was chairman. 
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QUALITATIVE REVIEW OF TERMINAL OBJECTIVES IN READING 

William H. Melehing 



A fow years ago, a colleague and 1 set out to clarity the methods, terms, and criteria 
associated with behavioral objectives.' Bas(»d on our review of numerous sets of objec- 
tives {)repared in various instructional contexts, we concluded that the most useful kind 
of objective was the terminal objective. We defined a terminal objective as a statement of 
the behavior expected of a student in a life or work performance situation, A terminal 
objective referred to behavior performed for its own sake. Thus, for example, while one 
mijiht not copy letters of the alphabet in a life or work performance situation, he might 
construct words from such letters, especially if he arranged the words in a particu- 
lar secjuence. 

Because we saw great variation in the level at which objectives were stated, we 
suggested a m(»ans by which terminal objectives might be meaningfully classified and/or 
evaluated. \\v off(>red five classification factors, each with three levels. The factors and 
levels are shown in Figure 1. With th»s classification scheme, we felt a trained reviewer 
could readily evaluate a set of objectives. 
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Figure 1 



Recently, the New York State Education Department (NYSED) adopted our 
concept of the terminal objective. NYSED undertook a full-scale effort to produce 
terminal objectives in reading and hired Independent Learning Systems (ILS) to develop 
the objectives. ILS was directed to develop terminal objectives consistent with the 
definition we had provided. 



' Harry L. Ammt^man and William H. Molching. The Dcrivalion. Htxalysis and Classification of 
Inslruchnnal Objevtirvs. HumRRO Technical Report 06 4. May 1966. 
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In addition, NYSKD requested that the developer produce two types of terminal 
objectives— a minimal set and a maximal set. The minimal set referred to reading 
behaviors that are required to maintain economic independence in a modern society, and 
to fulfill one*s obligations as a parent and a citizen of the nation and community. By 
contrast, the maximal set referred to Heading behaviors that are necessary to meet the 
reading requirements of tasks representt^d by subject areas typically offered at the 
beginning of secondary school. 

A final requirement imposed on the developer by NYSED concerned the need to 
prepare a number of approximations (initially specified as four but later reduced) for 
each terminal objective. An approximation was defined by NYSED as ''an observable 
instance of behavior which is less complex, less difficult, requires more external support, 
or has a more limited range of manifestation relative to the terminal performance level. 
The approximation is further identifiable as the same* form or type of behavior that is 
specified in the terminal performance objective.** 

To ensure the production of acceptable objectives and to institute some quality 
control procedures, NYSED engaged HumRRO to review and evaluate the objectives 
according to the classification scheme mentioned above. This, in turn, gave us an 
opportunity to explore the feasibility and usefulness of the classification system. 

I have made no effort to sort our experiences into categories, nor have I attempted 
to prepare a complete summary of the results of the classification effort. Insti*ad I have 
electL^d to describe what we did, give a few results, and share with you some of the 
problems that we faced. To give you some idea of the scope of the review effort, I might 
mention that, excluding the objectives and approximations used in training sessions, a 
total of 312 terminal objectives and 766 approximations were evaluated by the raters. 

The Rating Procedure 

Three psychologists at the HumRRO Western Division served as raters. Each was a 
Ph.D. with some previous experience in preparing or evaluating objectives. None had used 
the Ammerman and Melching classification system, however. 

Each rater was given a copy of the Ammerman and Melching report and a copy of a 
brief paper on the definitions and descriptions of the levels of the five A&M factors. 
After raters had studied these materials, they were directed to independently rate a 
sample of 11 terminal objectives. These objectives were constructed so that each level of 
each factor was a candidate rating at least once. They then met with the principal 
investigator, discussed the ratmgs, and attempted to reconcile differences. Next, each 
rater was given 20 objectives prepared by ILS and asked to rate them independently. 
After this training, they were judged ready to perform their assigned functions. 

Verb Problems 

The first .set of objectives received for rating contained 87 objectives of the minimal 
set. Each objective was accompanied by four approximations. The objectives exhibited a 
faii'ly common feature— one that occasioned some distress in raters. This was the 
tendency for the **objective writer" to use multiple verbs to describe the desired student 
action. Sample objectives had occasionally possessed this feature, but this was not 
routine. Since the classification schem? assumes only one student action is stated, raters 
tried to handle the multiple verb objective by designating one verb as the prime one. 
However, they were not entirely successful. 

Here s a somewhat exaggerated example of an objective with multiple verbs: 

Given a copy of the XYZ Buying Guide, the student will use the 
index to locate a specified product and will state if the product is 
considered acceptable or unacceptable and will correctly list at least 
one reason given in the Guide for this rating. 



The inclination lor vriters to use> several verbs when writing reading objoctives may 
he peculiar to these kinds of objectives, but I doubt it. Let's look at the basic aspects of 
a reading objective. Presumably it requires the student to read a passage and then to 
behave in some way to demonstrate* comprehension of what he has read. So, for example, 
a student might be asked to read a label on a medicine bottle and then indicate his 
understandin^i of the label by answering a question like, *'What dosage should be given to 
a two-year old infant?" .An objective might require a student to perform two oi^^-^^re 
actions such as n>ad, locate, <'opy down, restate, and/or select. 

Writing objectives this way is due mostly to our previous writing habits. For 
convenience in time and effort, we tend to serialize student actions. Unfortunately, such 
duality (or multiplicity) creates problems when we attempt to evaluate performance. 
Which action shall we assess? Which action is critical? How many different student 
behaviors are pres(»nt? 

Since actions of "read,'* **use," and 'Mocate" are neither readily observable nor 
particularly informative, they might as well be deleted from statements. After all, if a 
student is given a written passage* that asks him to perform in some way, it is assumed 
that he caimot perform appropriately unless he has read it. What the writer must do 
when tempted to use multiple verbs is decide which student action is critical with respect 
to eom{)rehension of the written material. vSo, for example, instead of asking a student to 
"use the telephone lK)ok to locate the number of the fire department and place a call," 
ont* might state the objective as: 

Ciiv(»n an emergency situation requiring the services of the fire depart- 
ment, the student must, with the aid of the local telephone book, 
indicate (state) the correct number of the department. 

Whether he can successfully perform the actions of placing the call is a different item 
of behavior. 

Let's turn now to a related verb problem. This time, however, the focus is primarily 
on the relative specificity of the action verb. Because of earlier writing and speaking 
habits, tne objective writer may employ verbs that are too vague. I refer here to student 
action verbs such as ''respond correctly, reply correctly, comply with, abide by." Even 
'*asseitibU» correctly" can be quite vague— a fact known by every parent who has 
attempted to assemble a child's toy according to manufacturer's instructions. 

The only solution to this problem, of course, is to require the writer to employ 
more specific verbs. To do this, he must first determine explicitly just what the student is 
expected to do. As a result, there is generally both an increase in student action 
specificity and a decrease in the tendency to use multiple verbs in a single objective. 

Classification Results 

Table 1 shows the distribution of ratings on the first batch of objectives.' 

The values in the table are consensus ratings. That is, if at least two raters agreed on 
a given level for a given factor of an objective, that was the recorded rating. When each 
rater sel(H*ted a du'ferent level for a factor, it was necessary for raters to re-examine their 
classifications and to reconcile differences. These efforts encouraged exposure of unique 
inte'^pretations and frequently promoted unanimity among judges. 

At least four values stand out in the table. First, all objectives were rated as 
generalized tasks (Factor A, Level 2). This is what one might logically expect of reading 
behaviors. Second, student action was judged as *'stated only" in 70% of the cases 



'To interpret the value.s in Table 1 and in subsequent tables, the reader should refer to the 
classification scheme shown in Fij^ure 1. 
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Table 1 

Classification of 87 Terminal Objectives*' 

Factor 



Level 



2 
3 



B 



0 
87 
0 



23 
2 
62 



84 
3 
0 



i 



76 
10 
1 



40 
30 
17 



^Thf? dpproxmuUions wrfM** not (:ldss>f«<»(l dccorcJinq to thf? A&M systrni InstiM(l, 
thfv werM >ub)UCted to ai.othrr kind of oval.j.ition, onr n )t rrU'ViirU to thi* jriciin locus 
of th»s crituiuo session. 



(Factor B, Lt»vel 3). Third, although raters experienced difficulty with verbs, they rated 
student action as highly relevant in almost every case (Factor C\ Level 1). Thus, the raters 
agreed with the developer that the reading behaviors stated in the objectives were relevant 
to the designated v/cia performance situation. Fourth, the vast majority of objectives 
were rated as fully C0in,)lete with respect to structural components (Factor I), Level 1). 
In terms of precisentss (*» components, howev(»r, the rat<»rs were not so generous. Th(»y 
felt that over half of the objectives could be improved in prei-ision. 

All of the evaluations cited, plus other more detailed feedback, were provided to the 
developer. In addition, of course, NYSED provided its own evaluation of the first batch 
of objectives to the developer. Armed with this information, the original objectives were 
scrapped, and a new attack on the pn»paration of obj(»ctives and approximations was 
undertaken. Within a short time, some 225 ti»rminal objectives were produced by th(* 
developer; 61 were of the minimal set and 164 were of the maximal s(»t. The number of 
approximations per objective now varied from two to four.' A total of 766 approxi- 
mations were prepared. 

Before giving some of the results of this n(»w classification (»ffort. some lomments 
may be made about the characteristics of the new obj(»ctives. With respt*ct to ust» of 
verbs, the new objectives and approximations were notably superior to the original on(»s. 
Statements now described one student action, and verbs describing highly specific action 
tended to be used. While raters experienc(»d little difficulty rating the minimal set 
objectives, this was not always the (*ase for maximal set objectives. For example, raters 
insisted on revising one cluster of 18 maximal set obj(»ctives dealing with **r(»levancy" and 
** motivation." They felt that the objectives, as written, assess(»d relevancy or attract ivt»- 
ness of the stimulus materials, not the capability of the student to read. 

In another cluster of 13 maximal s(»t objectives where the* student was required to 
**write in his own words a correct definition of a concept or thing," raters rejected the* 
terminal objective and recommended that the first approximation be stated as an 
objective. The raters felt that Ihe behavior required by the approximation was more 
relevant to the intended work performance situation than that required by the originally 
stated objective. A full statement of this approximation is as follows: 

Given a prose selection from a textbook that defines a concept or 
thing and a statement in which the concept or thing is used incor- 
rectly, the student will state a corrected version of the statement. 



' One objectivt* actually had fivi* approximations. 
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other resuUs of the classification effort support the contention that the minimal set 
objectives were easier to rate than the maximal set objectives. Before presenting some of 
these results, however, it should be noted that this time raters classified both termmal 
objectives and approximations. Thus, 991 statements were classified. As before, if at least 
two raters agreed on the level of a factor, that agreement was accepted and recorded. 
Only in instances where there was maximal disagreement among raters (i.e., each rater 
.selected a different level for a factor) was it necessary to have raters discuss their ratmgs 
and reconcile differences. 

Table 2 shows the classification results for the minimal set objectives. The most 
striking thing about these values is that the ratings on the first three factors show no 
variation. All 61 statements were viewed by the raters as being generalized tasks, partially 
described, and highly relevant to an intended work situation. By contrast, raters felt the 
completeness and precision of structural components of many of the objectives were not 
fully acceptable. 

Table 2 

Classification of 61 Minimal Set Terminal Objectives 



Factor 





A 


B 


c 


D 


E 


1 


0 


0 


61 


38 


23 


2 


61 


61 


0 


23 


38 


3 


0 


0 


0 


0 


0 



Now let's look at the classifications given to the maximal set terminal objectives. 
They appear in Table 3. Except for the Factor C rating, the classification pattern of the 
maximal set objectives is not very different from that of the minimal set objectives. The 
difference in the intended work situation for maximal objectives versus mmimal objec- 
tives probably accounts for the spread in relevancy seen in Table 3. 

Tables 4 and 5 show how the approximations of the minimal and maximal sets of 
obje' tives were classified. The classification pattern of the minimal set approximations is 
mor Uke that of the maximal sc l objectives than it is of the minimal set objectives. This 
is due primarily to the rating given to Factor C. Over one-third of the minimal set 



Table 3 

Classification of 164 Maximal Set Terminal Objectives^ 



Factor 



Level 





A 


B 


C 


D 


E 


1 


0 


14 


84 


94 


39 


2 


149 


144 


56 


64 


119 


3 


9 


0 


18 


0 


0 



^Because six of the objectives were viewed as identical to a seventh objective, 
they were deleted and not rated. 



ERIC 



5 

9 



Table 4 

Classification of 200 Minimal Set Approximations 



Factor 

» 



Level 





A 


B 


C 


D 


E 


1 


0 


1 


131 


135 


99 


2 


200 


199 


47 


65 


101 


3 


0 


0 


r 22 


0 


0 



Table 5 

Classification of 566 Maximal Set Approximations^ 



Factor 





A 


B 


C 


D 


E 


1 


0 


0 


159 


358 


239 


2 


492 


477 


318 


170 


289 


3 


36 


51 


51 


0 


0 



Thirteen of the approximations were judged to be better terminal objectives 
than the objectives originally stated, and they were not included in this table. Also, 
25 other approximations were not classified either because they were judged as incon- 
gruent with their associated objectives or because the objectives to which they belonged 
were deleted. 



approximations were judged as being only moderate or low in relevance. Table 2 shows 
that none of the minimal set objectives obtained these ratings; all were judged highly 
relevant in student action. 

The classification pattern of the maximal set approximations appears to be quite 
similar to the pattern of the maximal set objectives. Some minor variations can be found, 
but their meaning is questionable. 

Maximal Disagreement 

One final bit of information relates to the proportion of instances in which raters 
evidenced maximal disagreement in their judgments. A maximal disagreement was defined 
as any instance in which each rater assigned a different level for a given factor. (See 
Table 6.) 

Whether the first 116 objectives (and their 403 approximations) were infinitely more 
difficult than the remaining objectives, or whether there is simply a pronounced practice 
effect cannot be determined. But the drop in number of disagreements from start to 
finish is striking. Note too, that not a single maximal disagreement was recorded when 
raters were judging the minimal set objectives. 
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Tdhlf 6 

Incidence of Maximal Disagreement Among Raters 



S* ii*ffmf?n t 


I 

Numher <)J 
1 Stdternpn 


Nutt)t)Rr of 

Disaijffiement 
Oppof tunit les'' 


Actual 
Number of 
Disagreements 


Percent of 
Disagreements 


M<jy mal Obi»?(:tw*-*s 
F»rst Gr0vj[) 


116 


580 


68 


11.7 


Miix.mdl Ap)f)rOximdTjOns 
First Gtoup 


403 


2015 


368 


18.3 


Mm iTiHl Ohjectivf.'S 


61 


305 


0 


0.0 


Mjnimal Af)[)rox»niations 


200 


1000 


8 


.8 


Maximal Ok)i<*ctiv(*s 
Last Group 


48 


240 


1 


.4 


Maximal Approximations 
Last Grouf) 


160 


800 


2 


.3 



*' Smce each statement was ^ate(J on five factors, the number of opportunities for maximal disagreement was 
obttin^ft ' by multiplying the number ot statements in a group by five. 



Conclusions 

The following conclusions appear warranted: 

(1) Raters can learn to apply the A&M classification system, and extended 
practice increases their skill. 

(2) The act of classifying objectives is an effective way to examine the 
communicability of objectives. 

(3) The classification system provides useful information by which to evalu- 
ate objectives. 

(4) The verb is probably the most important aspect of an objective, especially 
in judging the relevancy of the learner*s behavior to the work performance situation. 

(5) Reading objectives are no more difficult to classify than are objectives in 
other instructional content areas. 
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